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Abstract: Designers have a limited selection of K-12 online course creation standards to choose 
from that are not blocked behind proprietary or pay walls. For numerous institutions and states, 
the use of the iNACOL National Standards for Quality Online Courses is becoming a widely used 
resource. This article presents the final phase in a three-part study to test the validity and reliability 
of the iNACOL standards specifically to online course design. Phase three was a field test of the 
revised rubric based on the iNACOL standards against current K-12 online courses. While the 
results show a strong exact match percentage, there is more work to be done with the revised 
rubric. 

Keywords: K-12 online learning, K-12 distance education, virtual school, cyber school, online 
course design 

Resume : Les concepteurs ont une selection limitee des normes K-12 de creation de cours en ligne a 
choisir qui ne sont pas bloques derriere des proprietes exclusives ou des peages informatiques. 

Pour de nombreuses institutions et etats, l'utilisation des Normes nationales pour les cours en ligne 
de qualite iNACOL devient une ressource largement utilisee. Cet article presente la phase finale 
d'une etude en trois parties pour tester la validite et la fiabilite des normes iNACOL 
specifiquement liees a la conception de cours en ligne. La phase trois etait une mise a Tessai sur le 
terrain de la rubrique revisee etablie en fonction des normes iNACOL par rapport aux cours en 
ligne K-12 actuels. Bien que les resultats montrent un fort pourcentage de correspondance exacte, il 
y a plus de travail a faire avec la rubrique revisee. 

Mots cles : iNACOL, rubriques, cours en ligne, qualite, K-12 

Introduction 

The use of online courses in the K-12 environment continues to grow, with supplemental online 
course enrollments at roughly 4.5 million in the United States alone (Gemin, Pape, Vashaw, & Watson, 
2015). This influx of online courses into the United States education system has led to a realization of 
the differences between traditional and virtual environments. These differences would include the 
issue of the design of asynchronous course content. However, what is somewhat surprising is that the 
research into this critical aspect of K-12 online learning has been both minimal (Barbour, 2013; Barbour 
& Adelstein, 2013a), and narrow in scope, mainly focusing on specific schools (Barbour, Morrison, & 
Adelstein, 2014; Friend & Johnston, 2005; Zucker & Kozma, 2003). 

There are current foundations and associations, such as the Michigan Virtual Learning Research 
Institute (MVLRI), that have taken up the task of researching further into course design. For example, 
since 2013 the MVLRI has included recommendations into educational delivery models and 
instructional design standards in their yearly directives for the Michigan Legislature (MVLRI, 2016). 

To date, the recommendations made by the MVLRI have focused primarily on the International 
Association for K-12 Online Learning's (iNACOL) (2011) National Standards for Quality Online Courses. 
At present, the iNACOL online course design standards are one of the most popular, non-proprietary 
and publically available standards - both in the United States and internationally. Yet, the iNACOL 
standards were not developed using a traditional process that examines the validity and reliability of 
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the standards and any instruments (i.e., rubrics) designed to measure those standards (Barbour, 2013; 
Barbour & Adelstein, 2013b; Molnar, Rice, Huerta, Shafer, Barbour, Miron, Gulosino, & Horvitz, 2014). 

The following article outlines the third, and final phase, of a research study designed to begin 
the process of examining the iNACOL online course design standards for validity and 
reliability. The first phase of research of this study provided a cursory review of the iNACOL 
standards to determine the level of support for each of the standard elements within the K-12 
online learning literature, as well as the broader online learning literature (^ee Adelstein & 

Barbour, 2016). During the second phase of this research study, two panels comprising eight 
experts from a variety of sectors in the field of K-12 online learning, examined the standards 
based on the outcome of phase one over a cycle of three rounds of review (^ee Adelstein & 

Barbour, in press). This second phase generated a revised list of specific design standards, as 
well as a revised rubric. In this article we describe the third phase of this research study, where 
four groups of two reviewers applied the phase two revised rubric using current K-12 online 
courses to examine the instrument for inter-rater reliability. 

Literature Review 

As indicated above, the research focused on K-12 online course design has been sparse. This 
can possibly be attributed to the idea that online course design has not been stressed in teacher 
professional development (Dawley, Rice, & Hinck, 2010; Rice & Dawley, 2007; Rice, Dawley, 

Gasell, & Florez, 2008). While it has been suggested that design should be a completely 
separate role from the classroom instructor (Davis, Roblyer, Charania, Ferdig, Harms, 

Compton, & Cho, 2007), this notion has only been promoted in a handful of models. For 
example, the Teacher Education Goes Virtual Schooling 1 and Supporting K-12 Online Learning 
in Michigan 2 programs focused primarily on the role of the online learning facilitator, while 
the Iowa Learning Online 3 and Michigan Online Teaching Case Studies 4 initiatives focused on 
the role of the online teacher. However, there are several design trends that can be gleaned 
from the available literature. The release of a variety of general design standards, practitioner- 
and advocacy-generated literature, and limited research, provide initial suggested guidance in 
online course design with enough commonalities to help form a larger picture, albeit one that 
is completed in broad strokes. 

The first theme in the literature focused on keeping navigation simple. The design of the 
course should be formatted in a way that allows for intuitive, easy navigation of the site. For 
example, course designers from the Centre for Distance Learning and Innovation (CDLI) used 
a template providing a consistency, so that it "doesn't frighten the kids with a different 
navigation menu on every screen" (Barbour, 2007a, p. 102). To add to the understanding, it 
was recommended that designers give students a tour of the course, explaining how the 
virtual classroom is organized (Elbaum, McIntyre, & Smith, 2002). When used by VHS, the 
majority of students agreed that the orientation gave them the comfort level to successfully 
navigate a course (Zucker & Kozma, 2003). This was also found to be important for students 
with special needs, as consistent navigation patterns curbed frustration (Keeler & Horney, 

2007). One of the positive aspects of courses with clarity and simplicity was that it not only 
worked for students with disabilities, but was also appropriate for all users (Keeler, Richter, 
Anderson-Inman, Horney, & Ditson, 2007). It was noted that a simplistic, linear approach 
should not necessarily bleed into content delivery, as a variety of activities allows for a more 
interesting course, as well as tapping into different student learning styles (Barbour 2007a; 

Elbaum, McIntyre, & Smith, 2002; Barbour & Cooze, 2004). 
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The second theme focused on less text and more visuals where appropriate. The use of a visual 
over text can offer advantages to students enrolled in an online course. The perception from 
educators that students ignore text-heavy sites plays into the notion that online courses are, 
and should be, presented differently than traditional courses (Barbour, 2007a). Online 
information may be presented in unique formats, and using solely text is akin to assigning a 
reading from the textbook (Barbour, 2005). It was therefore not surprising to see online 
educators ask for additional training so they could create and add multimedia into their 
courses (Barbour, Morrison, & Adelstein, 2014). Students agreed and indicated that they found 
visuals and multimedia, "really interesting and a lot better than sitting down and reading the 
book" (Barbour & Adelstein, 2013a, p. 60). A graphically intensive course also allows visual 
learners to flourish (Barbour & Cooze, 2004), as well as helping to provide structure to 
students with disabilities (Keeler et al., 2007). However, graphics should be used only when 
appropriate, and not just because they are readily available (Barbour, 2007; Elbaum, McIntyre, 
& Smith, 2002). Too many or over-stimulating visuals and backgrounds may distract students 
with attention deficit disorders (Keeler & Horney, 2007), which is why a mix of audio, text, and 
visuals is recommended. 

The third theme focused on clear instructions. The nature of online courses, especially 
asynchronous courses, means that clear and detailed directions are needed to help move 
students along (Elbaum, McIntyre, & Smith, 2002). For example, Barbour (2007a) indicated that 
"the directions and the expectations [need to be] precise enough so students can work 
effectively on their own, not providing a roadblock for their time" (p. 104). Clarity was also a 
concern for students, who worried that online content, was not as straightforward as the 
textbook, or that it was not easily accessible (Barbour & Adelstein, 2013a). In fact, the notion of 
clarity was relevant enough for VHS to include it as one of the 19 standards used for their 
course review process. The standards asked designers to judge if, "the course is structured in 
such a way that organization of the course and use of medium are adequately explained and 
accommodating to the needs of students" (Yamashiro & Zucker, 1999, p. 57). The use of 
consistent, explicit expectations was also important for exceptional students to stay on track as 
well (Keeler et al., 2007). The idea is that clarity of expectations will remove instructions as a 
possible barrier for students, allowing the student and instructor to focus on the learning. 

This leads into the final theme, which focused on providing feedback to students. Since the 
students do not have the ability to talk directly with the teacher in classroom as in a traditional 
course, it is important to provide frequent, reliable and predictable feedback (Elbaum, 
McIntyre, & Smith, 2002). As was the case with the previous suggestion, VHS reviewed 
courses with feedback in mind, checking that "the structure of the course encourages regular 
feedback" (Yamashiro & Zucker, 1999, p. 57). Feedback can be accomplished in a variety of 
ways, from self-assessments to built-in auto-graded exams found in certain learning 
management systems (Elbaum, McIntyre, & Smith, 2002). A self-assessment feature that gives 
instantaneous feedback, for example, was highly touted by online students (Barbour & 
Adelstein, 2013b), who appreciated knowing immediately if they were on the right track. 
Immediate feedback can be a beneficial formative assessment for students (Huett, Huett, & 
Ringlaben, 2011). Regardless of the form it takes, reliable feedback to students is vital to a 
course, as it keeps the students up-to-date on their progress and engaged in their learning 
(Elbaum, McIntyre, & Smith, 2002). 

The four principles outlined are a small but important collection of common elements of K-12 
course design literature. However, there is clearly more that should be taken into 
consideration for online delivery, which is the focus of the overall study. This article will focus 
on phase three, which looks to field test the revised rubric designed in phase two. The revised 
rubric contains elements determined to be vital by an expert panel in regards to specifically K- 
12 online course design. 
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Methodology 

Upon completion of phase one and two, which tested content validity through a comparison to 
the standards in the literature and then expert review, the third and final phase of this study 
examined the reliability of the rubric based on the revised iNACOL standards. When 
evaluating the rubric, it was important to test not just the validity, but the reliability as well 
(Taggart, Phifer, Nixon, & Wood, 2001). Further, Legon and Runyon (2007) noted that having 
instructors review online course design rubrics not only helped the instrument, but was also 
beneficial to the instructors. These instructors mentioned feeling stimulated and motivated to 
improve their own courses based on their learning from participation in the review process. 
Simply put, inter-rater reliability is a form of triangulation (Denzin, 1978), which is a method 
used to assess the accuracy of a specific point using different inputs. 

Inter-rater reliability for pairs of reviewers using multiple responses can be determined in 
different ways, with kappa being one of the more popular methods. Initially, the kappa 
coefficient appeared the most appropriate, as it, "indicates whether two judges classify entities 
in a similar fashion" (Brennan & Hays, 2007, p. 155). However, as the data was reviewed, it 
became obvious that using kappa would be impossible to accomplish. Kappa cannot be 
calculated if a rater gives the same rating to what is being tested, as the rater changes from a 
variable to a constant. Since the study took the details of each specific element into account, 
there was an increased likelihood of the same rating being applied by one or both reviewers 
(this issue is discussed in further detail in the results). Understanding the limitations of using 
such a small pool of results, the results were ultimately shared through percentage agreement. 
As noted by Neuendorf (2002), "coefficients of .90 or greater are nearly always acceptable, .80 
or greater is acceptable in most situations, and .70 may be appropriate in some exploratory 
studies for some indices" (p. 145 as cited by Moore, 2015, p. 26). 

The purpose of this phase of the study was to field test the revised rubric using online courses 
that were already in use by K-12 online learning programs. The reviewers were K-12 online 
designers and / or K-12 online instructors who were not involved with the second phase of this 
study (see Table 1). 

Table 1. Description of the Four Groups of Reviewers 


Group A 

Group B 

Bob (all names are pseudonyms) 

Ashley 

• High school educator with K-12 online 

• Secondary educator with online design 

experience 

experience 

• Midwest region 

• West Region 

Hilary 

Andrea 

• K-12 online educator 

• Educator and Administrator in K-12 online 

• Northeast Region 

education 


• West Region 
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Group C 


Group D 


Donald 

• High school educator with K-12 online 
experience 

• Midwest region 
Nancy 

• High school and online educator with design 
experience 

• West region 


Josh 

• Educator and Administrator in K-12 online 
education 

• West Region 
Sarah 

• Educator and Administrator in K-12 online 
education 

• West Region 


Designers and instructors were selected because they were representative of the population 
who would most likely use the newly revised rubric. The specific sample represented both a 
convenient and purposeful group of individuals. 

As the reliability of an instrument is actually improved upon when the users undergo training 
(Taggart et al., 2001), the groups were trained in the different areas of measurement as well as 
the use of the rubric. After each reviewer agreed to participate, they were sent a training 
packet that included the revised rubric, examples on how to grade specific elements, and a 
sample course to test the rubric against (see Appendix A for a copy of this training packet). 
One week later, a Google Hangout meeting was scheduled with each group to discuss the 
results of their application of the rubric to the sample course. 

Upon completion of the meeting, each group received five courses to review. Reviewers had 
up to two weeks to individually complete the process. Courses reviewed covered core 
academic areas, as well as electives for both middle school and high school, from two different 
online course providers 5 (see Tables 2 and 3). 


5 An application process to use a third provider to supply elementary courses for this phase of the study was completed, but 
ultimately the provider stopped communicating. Contact was attempted multiple times over the course of two months, but 
eventually the study moved forward without the third provider. 
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Table 2. Types of Courses Reviewed 


Grade 

Subject Matter 

Level 

Elective 

Language Arts 

Mathematics 

Science 

Social Studies 

6 

X 


X* 



7 

X 

X* 



X 

8 

X 





9 

X 




X* 

10 



X 

X 

X 

11 



X 

X* 



X* = Course was designed to fit within multiple areas of middle school (MS) or high school (HS). 


Table 3. Courses Reviewed by Groups 



Subject Matter/School Level 

MS 

Elect 

MS 

ELA 

MS 

Math 

MS Sci 

MS SS 

HS Elect 

HS ELA 

HS 

Math 

HS Sci 

HS SS 

Group A 

X 

X 






X 


x,x 

Group B 

X 




X 


X 

X 

X 


Group C 


X 

X 



x,x 



X 


Group D 


X 



X 

x,x 



X 



Each group used the revised rubric to review the five courses and rated the measurements on a 
three-point Likert type scale (see Appendix B). If the element was evident in the course it was 
rated a '3' for applied, a '2' was for elements that were partially applied, and a rating of 'l' 
meant the element was not applied. 

The results between group members were coded using three levels. According to Bresciani, 
Oakleaf, Kolkhorst, Nebeker, Barlow, Duncan, and Hickmott, (2009), if the rubric is well- 
designed, even untrained evaluators will find a significant level of agreement. As such, results 
were tabulated by the size of difference per rating, looking at 'exact match/ 'different by one/ 
and 'different by two.' Of particular importance were the exact matches as well as those that 
were different by two. In the latter situation, it would suggest that one reviewer in the group 
found no evidence of the element while the other believed that it was fully applied. 
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Results 

The results of the field test are presented by section titles as used in the revised rubric. 

Section A: Content 

Overall, Section A did not have strong consistency across the groups (see Table 4). 


Table 4. Section A Element Size Difference per Group 



1 

r 2 

Size of Difference 

Size of Difference 

Exact Match 

Different by One 

Different by Two 

Exact Match 

Different by One 

Different by Two 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

A1 

40.0% 

2 

20.0% 

1 

40.0% 

2 

20.0% 

1 

40.0% 

2 

40.0% 

2 

A2 

60.0% 

3 

40.0% 

2 

0.0% 

0 

20.0% 

1 

20.0% 

1 

60.0% 

3 

A3 

40.0% 

2 

0.0% 

0 

60.0% 

3 

20.0% 

1 

20.0% 

1 

60.0% 

3 

A4 

40.0% 

2 

60.0% 

3 

0.0% 

0 

60.0% 

3 

40.0% 

2 

0.0% 

0 

A5 

60.0% 

3 

40.0% 

2 

0.0% 

0 

40.0% 

2 

40.0% 

2 

20.0% 

1 

A6 

100.0% 

5 

0.0% 

0 

0.0% 

0 

20.0% 

1 

80.0% 

4 

0.0% 

0 

A7 

20.0% 

1 

20.0% 

1 

60.0% 

3 

20.0% 

1 

40.0% 

2 

40.0% 

2 

A8 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

A9 

60.0% 

3 

20.0% 

1 

20.0% 

1 

40.0% 

2 

20.0% 

1 

40.0% 

2 

3 

4 

Size of Difference 

Size of Difference 

Exact Match 

Different by One 

Different by Two 

Exact Match 

Different by One 

Different by Two 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

Row Valid 
N % 

Count 

A1 

100.0% 

5 

0.0% 

0 

0.0% 

0 

60.0% 

3 

20.0% 

1 

20.0% 

1 

A2 

80.0% 

4 

20.0% 

1 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

A3 

80.0% 

4 

20.0% 

1 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

A4 

20.0% 

1 

80.0% 

4 

0.0% 

0 

40.0% 

2 

0.0% 

0 

60.0% 

3 

A5 

40.0% 

2 

0.0% 

0 

60.0% 

3 

20.0% 

1 

80.0% 

4 

0.0% 

0 

A6 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

A7 

60.0% 

3 

40.0% 

2 

0.0% 

0 

80.0% 

4 

0.0% 

0 

20.0% 

1 

A8 

80.0% 

4 

20.0% 

1 

0.0% 

0 

40.0% 

2 

0.0% 

0 

60.0% 

3 

A9 

80.0% 

4 

0.0% 

0 

20.0% 

1 

80.0% 

4 

0.0% 

0 

20.0% 

1 


Still, more than half of the ratings were exact matches for groups one, three, and four. Two 
elements in particular, A6 (i.e., the course is free of bias) and A8 (i.e., privacy policies are 
stated), scored high - with 80% complete agreement across all groups (see Table 5). Taken as a 
whole across all groups. Section A had 58% complete agreement. 
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Table 5. Section A Size Difference Cross Tabulation All Groups 



Size of Difference 

Total 

.00 

1.00 

2.00 

A1 

Count 

11 

4 

5 

20 


% 

55.0% 

20.0% 

25.0% 

100.0% 

A2 

Count 

12 

5 

3 

20 


% 

60.0% 

25.0% 

15.0% 

100.0% 

A3 

Count 

12 

2 

6 

20 


% 

60.0% 

10.0% 

30.0% 

100.0% 

A4 

Count 

8 

9 

3 

20 


% 

40.0% 

45.0% 

15.0% 

100.0% 

A5 

Count 

8 

8 

4 

20 


% 

40.0% 

40.0% 

20.0% 

100.0% 

A6 

Count 

16 

4 

0 

20 


% 

80.0% 

20.0% 

0.0% 

100.0% 

A7 

Count 

9 

5 

6 

20 


% 

45.0% 

25.0% 

30.0% 

100.0% 

A8 

Count 

16 

1 

3 

20 


% 

80.0% 

5.0% 

15.0% 

100.0% 

A9 

Count 

13 

2 

5 

20 


% 

65.0% 

10.0% 

25.0% 

100.0% 


While every group had at least one element from Section A with 60% of the scores off by two, 
no specific element was off by two with a majority of the groups or all the groups. A3 
discussed having materials available at the course start and was flagged by two groups; while 
A4, A5, A7, and A8 each had one mention. 

Section B: Instructional Design Elements 

Much like Section A, there was not a notable consistency of exact agreement in Section B (see 
Table 6). 
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Table 6. Section B Element Size Difference per Group 



1 

’ - 2 

Size of Difference 

Size of Difference 

Enact Match 

Different by One 

Different by Two 

Enact Match 

Different by One 

Different by Two 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

B1 

40. OX 

2 

60. OX 

3 

o.ox 

0 

O.OX 

0 

100. OX 

5 

o.ox 

0 

B2 

60. OX 

3 

40. OX 

2 

o.ox 

0 

80. OX 

4 

20. OX 

1 

o.ox 

0 

B3 

60. OX 

3 

20. OX 

1 

20. OX 

1 

80. OX 

4 

20. OX 

1 

o.ox 

0 

B4 

40. OX 

2 

60. OX 

3 

O.OX 

0 

20. OX 

1 

60. OX 

3 

20. OX 

1 

B5 

60. OX 

3 

40. OX 

2 

O.OX 

0 

80. OX 

4 

20. OX 

1 

O.OX 

0 

B6 

80.0% 

4 

20.0% 

1 

0.0% 

0 

20.0% 

1 

80.0% 

4 

0.0% 

0 

B7 

80.0% 

4 

20.0% 

1 

0.0% 

0 

40.0% 

2 

60.0% 

3 

0.0% 

0 

B8 

60.0% 

3 

40.0% 

2 

0.0% 

0 

20.0% 

1 

80.0% 

4 

0.0% 

0 

B9 

40.0% 

2 

60.0% 

3 

0.0% 

0 

40.0% 

2 

60.0% 

3 

0.0% 

0 

BIO 

40. OX 

2 

60. OX 

3 

O.OX 

0 

80. OX 

4 

O.OX 

0 

20. OX 

1 

B11 

40. OX 

2 

60. OX 

3 

o.ox 

0 

20. OX 

1 

80. OX 

4 

O.OX 

0 

3 

4 

Size of Difference 

Size of Difference 

Enact Match 

Different by One 

Different by Two 

Enact Match 

Different by One 

Different by Two 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

B1 

100. OX 

5 

0.0X 

0 

o.ox 

0 

80. OX 

4 

20. OX 

1 

O.OX 

0 

B2 

80. OX 

4 

20. OX 

1 

O.OX 

0 

60. OX 

3 

40. OX 

2 

O.OX 

0 

B3 

60. OX 

3 

40. OX 

2 

O.OX 

0 

40. OX 

2 

60. OX 

3 

O.OX 

0 

B4 

20. OX 

1 

20. OX 

1 

60. OX 

3 

80. OX 

4 

20. OX 

1 

O.OX 

0 

B5 

100. OX 

5 

0.0X 

0 

O.OX 

0 

40.0% 

2 

20.0% 

1 

40.0% 

2 

B6 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

B7 

80.0% 

4 

20.0% 

1 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

B8 

100.0% 

5 

0.0% 

0 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

B9 

100.0% 

5 

0.0% 

0 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

BIO 

40. OX 

2 

0.0X 

0 

60. OX 

3 

40. OX 

2 

O.OX 

0 

60. OX 

3 

B11 

40. OX 

2 

60. OX 

3 

O.OX 

0 

40. OX 

2 

60. OX 

3 

O.OX 

0 


Three of the four groups once again had over a 50% exact match. Group two was again under 
50% for this section. Looking across all groups. Section B had a 57% exact match overall, yet 
none of the groups attained more than 75% on any given element (see Table 7). 
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Table 7. Section B Size Difference Cross Tabulation All Groups 




Size of Difference 




r 

.00 

1.00 

2.00 

Total 

B1 

Count 

11 

9 

0 

20 


% 

55.0% 

45.0% 

0.0% 

100.0% 

B2 

Count 

14 

6 

0 

20 


% 

70.0% 

30.0% 

0.0% 

100.0% 

B3 

Count 

12 

7 

1 

20 


% 

60.0% 

35.0% 

5.0% 

100.0% 

B4 

Count 

8 

8 

4 

20 


% 

40.0% 

40.0% 

20.0% 

100.0% 

B5 

Count 

14 

4 

2 

20 


% 

70.0% 

20.0% 

10.0% 

100.0% 

B6 

Count 

15 

5 

0 

20 


% 

75.0% 

25.0% 

0.0% 

100.0% 

B7 

Count 

10 

10 

0 

20 


% 

50.0% 

50.0% 

0.0% 

100.0% 

B3 

Count 

13 

7 

0 

20 


% 

65.0% 

35.0% 

0.0% _ 

100.0% 

B9 

Count 

13 

7 

0 

20 


% 

65.0% 

35.0% 

0.0% 

100.0% 

BIO 

Count 

10 

3 

7 

20 


% 

50.0% 

15.0% 

35.0% 

100.0% 

B11 

Count 

7 

13 

0 

20 


% 

35.0% 

65.0% 

0.0% 

100.0% 


There were significantly less 'different by two' counts for Section B. B4 attained the 60% 
threshold with one group. Only BIO, which discussed explicit communication, activities, and 
tools in the course at multiple intervals, had 60% of the scores separated by two numbers for 
more than one group. Overall, most of the elements fit into the exact match or one off. 

Section C: Student Assessment Elements 

The level of inter-rater reliability in Section C significantly improved compared to the prior 
two sections, with 'exact match' being the highest ranking for all four groups (see Table 8). 
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Table 8. Section C Element Size Difference per Group 



1 

- - 2 

Size of Difference 

Size of Difference 

Enact Match 

Different by One 

Different by Two 

Enact Match 

Different by One 

Different by Two 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Cl 

80.0% 

4 

20.0% 

1 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

C2 

40.0% 

2 

60.0% 

3 

0.0% 

0 

40.0% 

2 

60.0% 

3 

0.0% 

0 

C3 

40.0% 

2 

60.0% 

3 

0.0% 

0 

20.0% 

1 

80.0% 

4 

0.0% 

0 

C4 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

C5 

100.0% 

5 

0.0% 

0 

0.0% 

0 

40.0% 

2 

60.0% 

3 

0.0% 

0 

C6 

20.0% 

1 

60.0% 

3 

20.0% 

1 

80.0% 

4 

20.0% 

1 

0.0% 

0 

C7 

60.0% 

3 

40.0% 

2 

0.0% 

0 

40.0% 

2 

60.0% 

3 

0.0% 

0 


3 

4 

Size of Difference 

Size of Difference 

Enact Match 

Different by One 

Different by Two 

Enact Match 

Different by One 

Different by Two 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Cl 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

C2 

60.0% 

3 

40.0% 

2 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

C3 

100.0% 

5 

0.0% 

0 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

C4 

100.0% 

5 

0.0% 

0 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

C5 

60.0% 

3 

40.0% 

2 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

C6 

60.0% 

3 

40.0% 

2 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

C7 

40.0% 

2 

60.0% 

3 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 


Cl (i.e., consistency of student evaluations in regards to goals and objectives) and C4 (i.e., 
students are continuously aware of progress) were both at 95% exact match across all groups 
(see Table 9). 

Table 9. Section C Size Difference Cross Tabulation All Groups 




Size of Difference 





.00 

1.00 

V 2.00 

Total 

Cl 

Count 

19 

1 

0 

20 


% 

95.0% 

5.0% 

0.0% 

100.0% 

C2 

Count 

12 

8 

0 

20 


% 

60.0% 

40.0% 

0.0% 

100.0% 

C3 

Count 

12 

8 

0 

20 


% 

60.0% 

40.0% 

0.0% 

100.0% 

C4 

Count 

19 

1 

0 

20 


% 

95.0% 

5.0% 

0.0% 

100.0% 

C5 

Count 

14 

6 

0 

20 


% 

70.0% 

30.0% 

0.0% 

100.0% 

C6 

Count 

12 

7 

T 

20 


% 

60.0% 

35.0% 

5.0% 

100.0% 

C7 

Count 

12 

8 

0 

20 


% 

60.0% 

40.0% 

0.0% 

100.0% 
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Overall, the four groups came out with a 71% exact match agreement. Nearly all groups 
showed results that were an exact match or off by a score of one. C6, which looked for a 
suggested grading rubric, was the only element that had a pair of scores two apart. This only 
occurred once, with group one, out of twenty total reviews across all groups. 

Section D: Technology 

The results for Section D were both consistent and inconsistent in comparison to the other 
sections of the rubric. To start. Section D had high exact match agreements for all four groups 
(see Table 10). 

Table 10. Section D Element Size Difference per Group 



1 

- - 2 

Size of Difference 

Size of Difference 

Exact Match 

Different by One 

Different by Two 

Exact Match 

Different by One 

Different by Two 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

01 

0.0% 

0 

80.0% 

4 

20.0% 

1 

20.0% 

1 

20.0% 

1 

60.0% 

3 

D2 

60.0% 

3 

40.0% 

2 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

D3 

100.0% 

5 

0.0% 

0 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

D4 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

D5 

80.0% 

4 

20.0% 

1 

0.0% 

0 

20.0% 

1 

60.0% 

3 

20.0% 

1 

D6 

80.0% 

4 

20.0% 

1 

0.0% 

0 

20.0% 

1 

80.0% 

4 

0.0% 

0 

D7 

40.0% 

2 

20.0% 

1 

40.0% 

2 

100.0% 

5 

0.0% 

0 

0.0% 

0 

D8 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

D9 

80.0% 

4 

0.0% 

0 

20.0% 

1 

100.0% 

5 

0.0% 

0 

0.0% 

0 

DIO 

100.0% 

5 

0.0% 

0 

0.0% 

0 

0.0% 

0 

0.0% 

0 

100.0% 

5 

3 

4 

Size of Difference 

Size of Difference 

Exact Match 

Different by One 

Different by Two 

Exact Match 

Different by One 

Different by Two 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

Row Valid 

NX 

Count 

D1 

20.0% 

1 

0.0% 

0 

80.0% 

4 

60.0% 

3 

20.0% 

1 

20.0% 

1 

D2 

60.0% 

3 

20.0% 

1 

20.0% 

1 

100.0% 

5 

0.0% 

0 

0.0% 

0 

D3 

80.0% 

4 

20.0% 

1 

0.0% 

0 

80.0% 

4 

20.0% 

1 

0.0% 

0 

D4 

60.0% 

3 

40.0% 

2 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

D5 

40.0% 

2 

20.0% 

1 

40.0% 

2 

80.0% 

4 

0.0% 

0 

20.0% 

1 

D6 

40.0% 

2 

0.0% 

0 

60.0% 

3 

80.0% 

4 

20.0% 

1 

0.0% 

0 

D7 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

D8 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

D9 

60.0% 

3 

0.0% 

0 

40.0% 

2 

80.0% 

4 

20.0% 

1 

0.0% 

0 

DIO 

100.0% 

5 

0.0% 

0 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 


For example, element D8, which discussed clearly stated copyright status, was an exact match 
for all 20 sets of reviews (see Table 11). Seven of the elements had at least a 75% exact match 
agreement across the groups, putting Section D at 81% overall agreement, the highest level for 
any section. 
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Table 9. Section D Size Difference Cross Tabulation All Groups 




Size of Difference 





.00 

1.00 

r 2.00 

Total 

Dl 

Count 

5 

6 

9 

20 


% 

25.0% 

30.0% 

45.0% 

100.0% 

D2 

Count 

16 

3 

1 

20 


% 

80.0% 

15.0% 

5.0% 

100.0% 

D3 

Count 

17 

3 

0 

20 


% 

85.0% 

15.0% 

0.0% 

100.0% 

D4 

Count 

18 

2 

0 

20 


% 

90.0% 

10.0% 

0.0% 

100.0% 

D5 

Count 

TT 

5 

4 

20 


% 

55.0% 

25.0% 

20.0% 

100.0% 

D6 

Count 

FT 

6 

3 

20 


% 

55.0% 

30.0% 

15.0% 

100.0% 

D7 

Count 

17 

1 

2 

20 


% 

85.0% 

5.0% 

10.0% 

100.0% 

DS 

Count 

20 

0 

0 

20 


% 

100.0% 

0.0% 

0.0% 

100.0% 

D9 

Count 

16 

T 

3 

20 


% 

80.0% 

5.0% 

15.0% 

100.0% 

DIO 

Count 

15 

0 

5 

20 


% 

75.0% 

0.0% 

25.0% 

100.0% 


However, Section D also had a high percentage of 'different by two' scores in the individual 
groups. For example, element DIO, which discusses the course following Family Educational 
Rights and Privacy Act (FERPA) regulations and posting the information, was at 100% 
disagreement in group two. Group three had 80% disagreement in regards to Dl, the element 
that relates to the course architecture allowing the instructor to add content, activities, and 
assessments. Looking across all the groups, Dl was at 45% with a score size difference of two. 

Section E: Course Evaluation and Support Elements 

With the lowest element count. Section E also had the lowest exact match scores (see Table 12). 
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Table 12. Section E Element Size Difference per Group 



1 

- - 2 

Size of Difference 

Size of Difference 

Enact Match 

Different by One 

Different by Two 

Enact Match 

Different by One 

Different by Two 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

El 

0.0% 

0 

100.0% 

5 

0.0% 

0 

0.0% 

0 

0.0% 

0 

100.0% 

5 

E2 

60.0% 

3 

40.0% 

2 

0.0% 

0 

20.0% 

1 

60.0% 

3 

20.0% 

1 

E3 

100.0% 

5 

0.0% 

0 

0.0% 

0 

0.0% 

0 

20.0% 

1 

80.0% 

4 


r 3 

4 

Size of Difference 

Size of Difference 

Enact Match 

Different by One 

Different by Two 

Enact Match 

Different by One 

Different by Two 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

Row Valid 

MX 

Count 

El 

0.0% 

0 

0.0% 

0 

100.0% 

5 

80.0% 

4 

20.0% 

1 

0.0% 

0 

E2 

20.0% 

1 

20.0% 

1 

60.0% 

3 

100.0% 

5 

0.0% 

0 

0.0% 

0 

E3 

60.0% 

3 

0.0% 

0 

40.0% 

2 

60.0% 

3 

40.0% 

2 

0.0% 

0 


Only groups 1 and 4 had over 50% exact matches. Across all groups, element E3, making sure 
the course offers technical support and assistance to the students and instructor, had the 
highest exact match rating at 55% (see Table 13). 


Table 13. Section E Size Difference Cross Tabulation All Groups 




Size of Difference 




r 

r 

.00 

1.00 

2.00 

Total 

El 

Count 

4 

6 

10 

20 


% 

20.0% 

30.0% 

50.0% 

100.0% 

E2 

Count 

10 

6 

4 

:: 


% 

50.0% 

30.0% 

20.0% ~~ 

:: : 

E3 

Count 

11 

3 

6 

:: 


% 

55.0% 

15.0% 

30.0% 

... 


Overall, Section E had a 41% exact match agreement. While Groups A and D did not have any 
two-point size differences. Groups B and C proved troublesome. Both Groups B and C had 
100% two-point disagreement for El, the element that checked for multiple means of assessing 
course effectiveness. Group B also disagreed on E3, scoring 80% of reviews with a two-point 
size difference. 

Looking at the reviews as a whole. Groups A, C, and D were in exact agreement over 60% of 
the time (see Table 14), with group 4 at nearly 75%. 
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Table 14. Overall Size Difference per Group 



r 

Size of Difference 

.oo r i.oo r 

2.00 

Total 

F 1 

Count 

123 

62 

15 

200 


% 

61.5% 

31.0% 

7.5% 

100.0% 


Count 

95 

71 

34 

200 


% 

47.5% 

35.5% 

17.0% 

100.0% 

5 

Count 

136 

32 

32 

200 


% 

66.0% 

16.0% 

16.0% 

100.0% 

r 4 

Count 

149 

35 

16 

200 


% 

74.5% 

17.5% 

8.0% 

100.0% 

Total 

Count 

503 

200 

97 

800 


% 

62.9% 

25.0% 

12T% 

100.0% 


Group B, however, was under 50% exact agreement. Group B also had the highest two-size 
difference, sitting at 17%. There are many reasons why Group B was so far off in exact 
matches, including personal bias or inadequate training from the principle researcher. If exact 
match were only taken into account, the 62.9% would not be acceptable for reliability. 

Discussion 

The overall results had numerous outcomes where there was a difference of two between the 
scores of the two reviewers. Many of the elements did not feed into opinion and bias (e.g., 
appropriate course rigor, high variety of learning pathways), but rather were based on 
whether the item was present or not (e.g., FERPA laws are posted, privacy policies). This 
would imply muddled course navigation, with some reviewers unable to find important 
course items. To help negate confusion, designers may use a standard template for their 
courses, much like those implemented at CDLI (Barbour, 2007b). CDLI designers insisted that 
navigation should be simple and minimal to avoid confusion (Barbour, 2007a). A basic 
document, with all the navigational procedures and important document locations outlined, 
for example, that would also be beneficial for students and instructors (Elbaum et al., 2002). 
Another option for a course would be to utilize unit checklists of expectations and indicate 
which should be effectively communicated to relevant stakeholders, including students (Huett 
et al., 2011). 

On the other hand, there were yes /no or simple direction elements (e.g., use of copyright 
materials) that were close to a 100% exact match. These elements were able to show proper 
modeling of how to apply the element in a clear and easy to understand fashion. The use of 
proper modeling is important for a course, since this is a concern not just for instructors but 
also for the students (Barbour & Adelstein, 2013a). When expectations are modeled correctly, it 
helps to remove the guesswork behind the meaning (Barbour, 2007a). Explicit expectations and 
modeling can extend to having a pacing guide that provides a clear overview of the 
requirements (Huett et al., 2011), which can have a positive impact on all students - including 
exceptional learners (Keeler et al., 2007). 

The elements that discussed use of appropriate multimedia and technology had some of the 
highest exact match scores, implying that the use of visual cues made rating the elements 
easier. Due to the unique medium of online courses, media should be used to enhance the 
course (Barbour, 2005; Barbour, 2007a). Courses that take advantage of multimedia foster 
student engagement (Barbour & Adelstein, 2013a), while those without multimedia may be 
bland and may make it difficult to sustain student interest (Huett et al., 2011). Notably, the 
overuse of multimedia can be a negative (Keeler & Horney, 2007), causing overstimulation. 
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However, when used appropriately multimedia visuals can offer structure for students (Keeler 
et al., 2007). Unfortunately, the ability to design various media elements is one of the most 
commons aspects that online teachers identified as requiring specific professional 
development (Barbour, Morrison, & Adelstein, 2014; Dawley, Rice, & Hinck, 2010). 

Conclusion and Implications 

The iNACOL (2011) National Standards for Quality Online Courses were compared to current 
literature in phase one (see Adelstein & Barbour, 2016), while an expert panel helped redesign 
a revised rubric that looked specifically at the course design standards (see Adelstein & 

Babour, in press). Phase three had K-12 online educators and course designers apply the 
revised rubric to existing online courses. Four teams of two applied the rubric to five courses 
each, which allowed the researcher to review the rubric for percentage agreement. This 
allowed the researcher to test the inter-rater reliability of the revised rubric. While the overall 
results do not meet a reliability threshold, there are still lessons to take away from the initial 
field test. The number of instances where there was agreement (i.e., 62.9%) or a differences of 
only one (i.e., 25%), strongly outweighed the number of instances where the reviewers had a 
difference of two (i.e., 12.1%). There are individual elements throughout the rubric that met the 
reliability threshold (i.e., 90% or 80%), while other elements may need to be revised and / or 
improved. Other considerations, such as bias or elements that were difficult to determine (e.g., 
course rigor, course assessment), need to be taken into account for the next revision. Overall, 
the revised rubric provided a narrow focus on course design elements only, which reinforced 
ideas that are currently promoted in K-12 online education. 

To discover the full potential of the revised rubric, further field tests are required to address 
the limitation of this initial study. One of the limitations was the small number of participants, 
which limited how inter-rater reliability could be calculated. Adding additional courses for 
each group, as well as expanding the number of groups, would allow for more reliable results. 
Another limitation was the use of the revised rubric with existing courses. While using existing 
courses was an appropriate place to begin the study, a true test would be to design multiple 
new courses utilizing the revised rubric. This would allow for future studies to compare 
designer and student opinions between courses created using the revised rubric with courses 
created using other standards. 


References 

Adelstein, D., & Barbour, M. K. (2016). Building better courses: Examining the construct validity of the iNACOL 
national standards for quality online courses. Journal of Online Learning Research , 2(1), 41-73. Retrieved from 
https: / / www.learntechlib.org/ d/171515 

Barbour, M. K. (2005). The design of web-based courses for secondary students. Journal of Distance Learning , 9(1). 

27-36. Retrieved from http: / / journals.akoaotearoa.ac.nz/ index.php/lOFDL/ article/view/122 
Barbour, M. K. (2007a). Principles of effective web-based content for secondary school students: Teacher and 
developer perceptions. Journal of Distance Education , 21(3), 93-114. Retrieved from 
http: / / www.jofde.ca/ index.php/jde/ article/ view/30 
Barbour, M. K. (2007b). Portrait of rural virtual schooling. Canadian Journal of Educational Administration and 
Policy, 59. Retrieved from http: / / www.umanitoba.ca/publications/cjeap/articles/barbour.html 
Barbour, M. K. (2013). The landscape of K-12 online learning: Examining what is known. In M. G. Moore (Eds.), 
Handbook of distance education (3 rd ed.) (pp. 574-593). New York: Routledge. 

Barbour, M. K., & Adelstein, D. (2013a). High-school students' perceptions of effective online course design. The 
Morning Watch, 41(1-2), 56-65. Retrieved from 

http: / / www.mun.ca/ educ/faculty/ mwatch/ vol41/fall2013/ michaelBarbour.pdf 


16 



Barbour, M. K., & Adelstein, D. (2013b). Voracious appetite of online teaching: Examining labour issues related to K-12 
online learning. Vancouver, BC: British Columbia Teachers Federation. Retrieved from 
http:/ / www.bctf.ca/uploadedFiles/Public/Issues/Technology/VoraciousAppetite.pdf 

Barbour, M. K., & Cooze, M. (2004). All for one and one for all: Designing web-based courses for students based 
upon individual learning styles. Staff and Educational Development International 8(2/ 3), 95-108. 

Barbour, M. K., Morrison, J., & Adelstein, D. (2014). The forgotten teachers in K-12 online learning: Examining 
the perceptions of teachers who develop K-12 online courses. International Journal of Online Pedagogy and 
Course Design , 4(3), 18-33. 

Brennan, P. F., & Hays, B. J. (1992). Focus on psychometrics: The kappa statistic for establishing interrater 
reliability in the secondary analysis of qualitative clinical data. Research in Nursing & Health, 15(2), 153-158. 

Bresciani, M., Oakleaf, M., Kolkhorst, F., Nebeker, C., Barlow, J., Duncan, K., & Hickmott, J. 

(2009). Examining design and inter-rater reliability of a rubric measuring research quality across multiple 
disciplines. Practical Assessment, Research & Evaluation, 14(12). Retrieved from 
http: / / www.pareonline.net/getvn.asp?v=14&n=12 

Davis, N., Roblyer, M. P., Charania, A., Ferdig, R., Harms, C., Compton, L. K. L., & Cho, M. O. (2007). 

Illustrating the "virtual" in virtual schooling: Challenges and strategies for creating real tools to prepare 
virtual teachers. The Internet and Higher Education, 10(1), 27-39. 

Dawley, L., Rice, K., & Hinck, G. (2010). Going virtual! 2010: The status of professional development and unique needs 
of K-12 online teachers. Boise ID: Boise State University. Retrieved from 
https: / / edtech.boisestate.edu/ goingvirtual/ goingvirtual3.pdf 

Denzin, N. K. (1978). The research act: A theoretical introduction to sociological methods. New York: McGraw Hill. 

Elbaum, B., McIntyre, C., & Smith, A. (2002). Essential elements: Prepare, design, and teach your online course. 
Madison, WI: Atwood Publishing. 

Friend, B., & Johnston, S. (2005). Florida virtual school: A choice for all students. In Z. L. Berge & T. Clark (Eds.), 
Virtual schools: Planning for success (pp. 97-117). New York: Teachers College Press. 

Gemin, B., Pape, L., Vashaw, L. & Watson, J. (2015). Keeping pace with K-12 digital learning: An annual review of 
policy and practice, 2015. Durango, CO: Evergreen Education Group. Retrieved from 
http:/ /www.kpkl2.com/wp-content/uploads/Evergreen KeepingPace 2015.pdf 

Huett, K. C., Huett, J. B., & Ringlaben, R. (2011). From bricks to clicks: Building quality K-12 online classes 
through an innovative course review project. Online Journal of Distance Learning Administration, 14(4). 
Retrieved from http:/ / www.westga.edu/~distance/ojdla/winterl44/huett huett ringlaben.html 

Keeler, C. G., & Horney, M. A. (2007). Online course designs: Are special needs being met? American Journal of 
Distance Education, 21(2), 61-75. 

Keeler, C., Richter, J., Anderson-Inman, L., Horney, M., & Ditson, M. (2007). Exceptional learners: Differentiated 
instruction online. In C. Cavanaugh & R. L. Blomeyer, (Eds.), What works in K-12 online learning (pp. 125-141). 
Eugene, OR: International Society for Technology in Education. 

Legon, R., & Runyon, J. (2007). Research on the impact of the quality matters course review process. In 23rd 
Annual Conference on Distance Teaching & Learning (pp. 8-10). Madison, WI: Division of Continuing Studies, 
University of Wisconsin-Madison. 

Michigan Virtual Learning Research Institute. (2016). 2015-16 directives support and accelerate innovation in online 
and blended learning. East Lansing, MI: Author. Retrieved from http: / /mvlri.org/About-Us/2015-16- 
Directives 

Molnar, A. (Ed.); Rice, J. K., Huerta, L., Shafer, S. R., Barbour, M. K., Miron, G., Gulosino, C, Horvitz, B. (2014). 
Virtual schools in the U.S. 2014: Politics, performance, policy, and research evidence. Boulder, CO: National 
Education Policy Center. Retrieved from http: / /nepc.colorado.edu/publication/virtual-schools-annual-2014 

Moore, P. (2015). An employer toolkit for employee training and policies related to social media and crisis 

communications in health care organizations. Unpublished doctoral dissertation. Ball State University, Muncie, 
IN. 


17 



Neuendorf, K. A. (2002). The content analysis guidebook. Thousand Oaks, CA: Sage. 

Rice, K., & Dawley, L. (2007). Going virtual! The status of professional development for K-12 online teachers. Boise ID: 

Boise State University. Retrieved from https: / /edtech.boisestate.edu/goingvirtual/goingvirtuall.pdf 
Rice, K., Dawley, L., Gasell, C., & Florez, C. (2008). Going virtual! Unique needs and challenges of K-12 online 
teachers. Boise ID: Boise State University. Retrieved from 
https: / / edtech.boisestate.edu/ goingvirtual/ goingvirtual2.pdf 
Taggart, G., S., Phifer, S. J., Nixon, J. A., & Wood, M. (1998). Rubrics: A handbook for construction and use. 
Lancaster, PA: Technomic Publishing Co. 

Yamashiro, K., & Zucker, A. (1999). An expert panel review of the quality of Virtual High School courses: Final report. 
Arlington, VA: SRI International. Retrieved from 

http: / / www.thevhscollaborative.org/sites/default/files/public/vhsexprt.pdf 
Zucker, A., & Kozma, R. (2003). The Virtual High School: Teaching generation V. New York: Teachers College 
Press. 

Authors 

David Adelstein, Instructional Technology, Wayne State University. E-mail: dave.adelstein@gmail.com 

Michael K. Barbour is a Ph.D. candidate at the University of Georgia. A social studies teacher by training, 
having taught in the traditional classroom and virtual high school environments, Michael is interested in the 
use of virtual schools to provide learning opportunities to rural secondary school students. E-mail: 
mkbarbour@gmail.com 


Appendix A 
Overview 

The overall goal will be to create a revised K-12 online course design rubric based on the iNACOL 
National Standards for Quality Online Courses. The first phase of research was to review the 
iNACOL standards based on the research literature to determine the level of support within the K-12 
online learning literature, as well as the broader online learning literature. For the second phase, eight 
experts in the field from a variety of sectors examined the standards in regards to course design 
beginning with the results of phase one. This second phase resulted in a revised list of standards and 
a revised rubric. In the third phase, 3-5 teams of two reviewers on the application of the revised rubric 
from phase two. 

Volunteers Needed for Phase Three 

• Educators with K-12 online teaching experience required 

• Educators with online course design experience preferred 

Phase Three Outline 

• Three to five teams of two reviewers 

• Each reviewer will apply the rubric independently to the content of four or five online courses 

• The rubric will be measured using inter-rater reliability 
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Phase Three Training 

• Reviewers will receive the rubric and examples of a course review 

• Each team will have a practice course to apply the rubric 

• Using Google Hangout, each team and the researcher will review the practice course 

Phase Three Time Commitment 

• Each reviewer will have approximately two weeks to complete all four or five reviews 

• Estimated time to review one course is approximately 60-90 minutes 

• Estimated total time to review all courses is four to eight hours 

Items to Keep in Mind 

• It is asked that you do not share information regarding the revised rubric, results, courses, or 
any other aspect of the research 

• Please contact the PI with any questions regarding training, commitment, or the review 
process 

Sample Application of Rubric 

While the main training will be come in the form of a group course review, this area will show 
examples of how to use the rubric. 


Example #1: Alignment with specific standards 

The first element in the revised rubric lists the following: 


SECTION A: CONTENT 

Element 

Further Explanation 

Subsection: Academic Content Standards and Assessments 

A1: The course content and assignments are 
aligned with the state’s content standards, 
common core curriculum, or other accepted 
content standards set for Advanced Placement 
courses, technology, computer science, or other 
courses whose content is not included in the 
state standards. 

The content and assignments for the core 
courses are explicitly and thoroughly aligned to 
the credit granting state’s academic standards, 
curriculum frameworks and assessments. 
Advanced Placement s courses must be 
approved with the College Board and other 
elective courses should be aligned to other 
nationally accepted content standards such as 
computer science, technology courses, etc. 

A1 Rating (1 = not applied, 2 = partially applied, 3 = applied): 

!□ 

_ 


Logging into the course up for review, there is an area labeled Standard Alignment. Looking further, 
the LMS shows how each of the course areas line up with the state standard ID code: 
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Grade Level 

State ID 

9 

N-Q.1. 

9 

N-Q.2. 

9 

N-Q.1. 

9 

N-Q.2. 

9 

A-CED.2. 

9 

A-CED.3. 

9 

F-BF 1(a) 

9 

A-CED.2. 

9 

A-REI.10. 

9 

F-IF.4. 

9 

F-IF.7(a) 

9 

N-Q.1. 

9 

F-IF.1. 

9 

F-IF.2. 

9 

F-BF. 4(a) 

9 

F-BF. 4(b) 

9 

F-BF. 4(d) 

9 

F-IF.2. 

9 

F-IF.4. 

9 

F-LE.5. 

9 

F-IF.1. 

9 

F-IF.4. 

9 

F-IF.5. 

9 

F-IF.1. 

9 

F-IF.4. 

9 

F-BF. 1(a) 

9 

F-BF.2. 

9 

F-IF.3. 


In this example, element A1 would receive a score of 3 (applied). 

Example #2: A complete overview and syllabus 

Some elements will have multiple items to look for, such as the one listed below: 


A4: A complete course overview and syllabus, 
which clearly states course goals and objectives, 
are included. Course goals are consistent with 
course requirements and are measurable in 
multiple ways. 

Within the learning management system the 
syllabus and overview objectives are present, 
explicitly stated, and can be easily found by 
students. The syllabus and overview objectives 
include: course objectives and student learning 
outcomes; assignments; student expectations; 
time requirements; required materials; the 
grading policy; teacher-student, teacher-parent 
contact policies; the intended audience; and the 
content scope and sequence. 

A4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 

n 

_ 


Looking at the course in question, there is an included syllabus that clearly states course goals and 
objectives: 


Course Objectives 

Throughout the course, you will meet the following goals: 


Reviewing the individual lessons, the goals are consistent with the requirements. However, when 
discussing a complete course overview, there are missing areas. The syllabus does not cover 
communication and contact policies. It might seem logical to have the teacher in charge of the course 
add this information on their own but as a reviewer you are strictly looking to see if the element was 
applied as listed. 
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In this example, element A4 would receive a 2 (partially applied). 

Example #3: Privacy policies 

Element A8 asks if privacy policies are posted: 


A8: Privacy policies are clearly stated. 

A policy statement is posted on the course 
provider’s website and/or in the learning 
management system disclosing the 
organization’s information gathering and 
dissemination practices. 

A8 Rating (1 = not applied, 2 = partially applied, 3 = applied): 




Looking through the included documentation and the course itself, it is clear that the privacy policy is 
not listed. However, the element text mentions that the policy can be listed on the course provider's 
website. As mentioned in example two, your review is limited to strictly the course and the areas of 
the learning management system (LMS) you have access to. You will not have access to the provider's 
website. 

In this example, element A8 would receive a 1 (not applied). 

Example #4: Students' needs and a variety of ways to learn 

There are elements that look at concepts that are slightly more abstract. A variety of methods, 
materials, and assessments is not as concrete as locating a syllabus. For example: 


B1: Course design reflects a clear 
understanding of all students’ needs and 
incorporates varied w avs to learn and master 
the curriculum. 

A variety of instructional and assessment 
methods, materials and assessments are 
used throughout the course, w hich allow 
students to demonstrate their achievement 
of the goals and objectives of the course. 

B1 Rating (1 = not applied, 2 = partially applied, 3 = applied): 

n 

i 


Each unit in the course includes a warm-up, instruction, summary, assignment, and quiz. While each 
unit does show a variety of learning methods (interactive assignments, listening comprehension, 
reading, etc.), the format does repeat for all units. The assessment methods utilize multiple choice for 
the majority of the quizzes and exams during this course. 

In this example, B1 would receive a 2 (partially applied). 
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Example #5: Communication opportunities 

In some instances, you will need to look outside the course at the LMS itself to review the element: 


Element 

Further Explanation 

Bll: The course provides opportunities (e.g. 
student-instructor, student-student interaction, 
student-course content, student-LMS) for 
masters and application of the material 

Learning activities and other learning 
opportunities are developed to foster 
student-instructor, student-student, and 
student-LMS interaction. The technologs' 
and course content encourage exchanges 
amongst the instructor and students 
through email, discussions, synchronous 
chats, simulations, lab activities and other 
group projects. Within the grading policy, 
guidelines defining student participation 
and expectations are provided. 

Threaded and/or synchronous discussions 
are available for developing community, 
asking and finding answers to questions 
about the course, and around the content. 
Access is available to groups or individual 
students based on the purpose of the 
activity. Rules, roles, and expectations for 
the discussion are clear and posted within 
the discussion forum. 

Bll Rating (1 = not applied, 2 = partially applied, 3 = applied): 


While the course does not have student-student or student-instructor interaction, the LMS does offer 
a communications area where email, group discussions, and chats can be set up by the instructor. 

In this example, Bll would receive a 3 (applied). 
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Appendix B 


SECTION A: CONTENT 

Element 

Further Explanation 

Subsection: Academic Content Standards and Assessments 

A 1 : The course content and assignments are 
aligned with the state's content standards, common 
core curriculum, or other accepted content 
standards set for Advanced Placement courses, 
technology, computer science, or other courses 
whose content is not included in the state 
standards. 

The content and assignments for the core courses 
are explicitly and thoroughly aligned to the credit 
granting state’s academic standards, curriculum 
frameworks and assessments. Advanced 

Placement® courses must be approved with the 
College Board and other elective courses should be 
aligned to other nationally accepted content 
standards such as computer science, technology 
courses, etc. 

A1 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



A2: The course content and assignments are of 
sufficient rigor, depth and breadth to teach the 
standards being addressed. 

The course components (objectives, assessments, 
instructional strategies, content, assignments and 
technology) are sufficiently broad, deep and 
rigorous such that successful students will have the 
knowledge and skills required by the standards 
upon completion of the course. 

A2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



A3: All course materials are available to students at 

course start. 

Before the course begins, students are provided 
learning resources that are utilized during the 
online course. These could include textbooks, 
instructional materials links to browser plugins, 
and other software, which students must install. 

A3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Course Overview and Introduction 

A4: A complete course overview and syllabus, 
which clearly states course goals and objectives, 
are included. Course goals are consistent with 
course requirements and are measurable in multiple 
ways. 

Within the learning management system the 
syllabus and overview objectives are present, 
explicitly stated, and can be easily found by 
students. The syllabus and overview objectives 
include: course objectives and student learning 
outcomes; assignments; student expectations; time 
requirements; required materials; the grading 
policy; teacher-student, teacher-parent contact 
policies; the intended audience; and the content 
scope and sequence. 

A4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION A: CONTENT 

Element 

Further Explanation 

A5: Information is provided to students, parents 
and mentors on how to communicate with the 
online instructor and course provider. 

Instructor information is provided to students with 
contact, availability, and biographical information. 
Information on how to contact the instructor via 
phone, email, and/or online messaging tools is 
provided within the contact information. If regular 
contact with the instructor is required as part of the 
course, clear expectations for meeting this 
requirement are posted within the course. 

A5 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Legal and Acceptable Use Policies 

A6: The course reflects multi-cultural education, 
and the content is accurate, current and free of bias 
or advertising. 

The course creates equal educational opportunities 
for students from diverse racial, ethnic, social-class 
and cultural groups. The content is up to date, 
accurate and free of any bias. 

A6 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



A7: Expectations for academic integrity, use of 
copyrighted materials, plagiarism and netiquette 
(Internet etiquette) regarding lesson activities, 
discussions, and e-mail communications are clearly 
stated. 

A “Code of Conduct” including netiquette 
standards, copyright and academic integrity 
expectations is provided. 

A7 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



A8: Privacy policies are clearly stated. 

A policy statement is posted on the course 
provider’s website and/or in the learning 
management system disclosing the organization’s 
information gathering and dissemination practices. 

A8 Rating (1 = not applied, 2 = partially applied, 3 = applied): 




SECTION A: CONTENT 

Element 

Further Explanation 

Instructor Resources 

A9: Online instructor resources (e.g. 
assessment, assignment answers and 
explanations, notes) are included. Pedagogy 
behind the resources are shared with 
instructors. 

Resources and notes, including assessments and 
access to answers, explanations to aid online 
instructors in teaching and facilitating the course 
are included within the learning management 
system. 

A9 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION B: INSTRUCTIONAL DESIGN ELEMENTS 

Element 

Further Explanation 

Subsection: Instructional and Audience Analysis 

Bl: Course design reflects a clear understanding of 
all students' needs and incorporates varied ways to 
learn and master the curriculum. 

A variety of instructional and assessment 
methods, materials and assessments are used 
throughout the course, which allow students to 
demonstrate their achievement of the goals and 
objectives of the course. 

Bl Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Modules and Resources 

B2: The course is organized by modules. Course 
design provides students with resources (e.g. 
alternate assignments, multimedia, simulations) 
that enrich course content. Each module includes 
an overview of the key objectives that incorporate 
a variety of activities, assignments, and resources 
to provide multiple learning opportunities for 
students to master the content. 

The course is organized by modules that fall 
into a logical sequence. At the start of each 
module, an overview is posted describing the 
activities, assignments, assessments, and 
resources to be used to complete the key 
objectives. A variety of activities, assignments, 
assessments, and resources are used to provide 
students with different paths to master the 
content. A wide variety of supplemental tools 
are clearly identified and readily available as 
well. 

B2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Instructional Strategies and Activities 

B3: The course instruction includes activities that 
engage students in active learning. 

The course provides multiple opportunities for 
students to be actively engaged in the content 
that includes meaningful and authentic 
learning experiences such as collaborative 
learning groups, student-led review sessions, 
games, analysis or reactions to videos, 
discussions, concept mapping, analyzing case 
studies, etc. 

B3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION B: INSTRUCTIONAL DESIGN ELEMENTS 

Element 

Further Explanation 

B4: The course provides options for instructors to 
adapt learning activities based on student needs, 
allowing for the course and instructors to offer 
learning paths that engage in a variety of ways. 

Students are given a variety of activities, 
assignments, assessments and resources to 
allow them to successfully master the content. 

If a student is unsuccessful with mastering a 
particular concept or is not challenged with the 
current module, the course content provides 
the instructor with suggestions they are able to 
use in order to provide additional remediation 
activities or alternative assignments. The 
instructor has access to adapt the course to 
meet the students’ needs by providing 
additional assignments, resources and activities 
for remediation or enrichments for the course. 

B4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



B5: The course provides opportunities for students 
to engage in higher-order thinking, critical 
reasoning activities and thinking in increasingly 
complex ways. 

Assignments, activities and assessments 
provide opportunities for students to elevate 
their thinking beyond knowledge and 
comprehension into the realm of analyzing 
situations, synthesizing information or 
evaluating an argument. Activities should 
include open-ended questions and encourage 
students to categorize and classify information. 
Opportunities for group work, decision-making 
and finding patterns should also be included in 
the course activities. 

B5 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



B6: Readability levels, written language 
assignments and mathematical requirements are 
appropriate for the course content and grade-level 
expectations. 

The course content should be written at 
appropriate readability levels for the grade 
level of the student audience and the grade 
level should be prominently explained within 
the course description. 

B6 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION B: INSTRUCTIONAL DESIGN ELEMENTS 

Element 

Further Explanation 

B7: The syllabus promotes a student plan of work 
with attainable expectations. 

The syllabus provides an academic outline for 
students in the course, which includes academic 
expectations at specific intervals. 

B7 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



B8: Activities are designed to encourage students' 
individual interests and goals. 

The course provides activities and assignments 
which are broad enough to allow for student 
connections. The connections are real world, such 
as personal interests, goals, or situations. 

B8 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Communication and Interaction 

B9: The course design provides opportunities for 
appropriate instructor-student interaction, including 
opportunities for timely and frequent feedback 
about student progress. 

Learning activities and other opportunities are 
created to foster instructor-student interaction. 
Students receive timely and frequent feedback 
on their progress that emphasizes the intended 
learner outcomes. The feedback is highly 
individualized, detailed, and recommends 
specific, individualized improvement, and 
strategies to encourage continued progress 
toward mastery. 

B9 Rating (1 = not applied, 2 ■ partially applied, 3 = applied): 



BIO: The course design includes explicit 
communication/activities/tools at multiple intervals 
throughout the course. The instructor confirms 
whether students are engaged and are progressing 
through the course. The instructor will follow 
program guidelines to address non-responsive 
students. 

Instructor-student interactions begin early 
enough in the course to confirm active 
participation by all students and continue 
throughout the course. 

BIO Rating (1 = not applied, 2 = partially applied, 3 = applied): 




SECTION B: INSTRUCTIONAL DESIGN ELEMENTS 

Element 

Further Explanation 

B11: The course provides opportunities (e.g. 
student-instructor, student-student interaction, 
student-course content, student-LMS) for mastery 
and application of the material. 

Learning activities and other learning 
opportunities are developed to foster student- 
instructor, student-student, and student-LMS 
interaction. The technology and course content 
encourage exchanges amongst the instructor 
and students through email, discussions, 
synchronous chats, simulations, lab activities 
and other group projects. Within the grading 
policy, guidelines defining student 
participation and expectations are provided. 


Threaded and/or synchronous discussions are 
available for developing community, asking 
and finding answers to questions about the 
course, and around the content. Access is 
available to groups or individual students 
based on the purpose of the activity. Rules, 
roles, and expectations for the discussion are 
clear and posted within the discussion forum. 

B11 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION C: STUDENT ASSESSMENT ELEMENTS 

Element 

Further Explanation 

Subsection: Evaluation Strategies 

Cl: Student evaluation strategies are consistent 
with course goals and objectives, are representative 
of the scope of the course and are clearly stated. 

The strategies used to assess students throughout 
the course are consistent with and aligned to what 
is presented in the course goals and objectives 
document posted within the course. 

Cl Rating (1 = not applied, 2 = partially applied, 3 = applied): 



C2: The course structure includes adequate and 
appropriate methods and procedures to assess 
students’ mastery of content. 

Assessment types are matched to the level of 
knowledge being tested. Both formative 
assessments (that inform and support learning) and 
summative assessments (that demonstrate mastery) 
are a part of the course structure. Student-selected 
assessment options, enabling learners to 
demonstrate mastery in different ways, are 
available. 

C2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Feedback 

C3: Ongoing and varied quality assessments 
aligned with course learning outcomes are 
conducted throughout the course to guide student 
instruction. 

The course provides quality and ongoing formative 
assessments to check for student understanding and 
to ensure they are prepared for the next lesson. 

Initial pre-tests may be provided to assess student 
readiness. 

C3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



C4: Assessment strategies and tools make the 
student continuously aware of his/her progress in 
class and mastery of the content. 

Feedback tools and procedures are built into the 
course to allow students to periodically self- 
monitor their academic progress. 

C4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Assessment Resources and Materials 

C5: Assessment materials provide the instructor 
with the flexibility to assess students in a variety of 
ways. 

Multiple versions of tests, test banks and other 
resources that support alternative evaluation 
methods are available. 

C5 Rating (1 = not applied, 2 = partially applied, 3 = applied): 




SECTION C: STUDENT ASSESSMENT ELEMENTS 

Element 

Further Explanation 

C6: Suggested grading rubrics are provided to the 
instructor. The instructor will share a chosen 
grading rubric with students. 

Rubrics, rationale, and/or characteristics are 
provided for each graded assignment. The 
instructor will make the final selection, which will 
then be shared with the students. 

C6 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



C7: The grading policy and practices are easy to 
understand and clearly communicated to students 
and parents. 

Grading policies and practices arc easy to read and 
clearly defined and may include any penalties that 
may be assessed to grades and/or extra credit 
opportunities. 

C7 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION D: TECHNOLOGY 

Element 

Further Explanation 

Subsection: Course Architecture 

D1: The course architecture permits the online 
instructor to add content, activities and assessments 
to extend learning opportunities where applicable. 

The instructor of record for the course has access to 
make additions to the content within the learning 
management system (LMS). Access should allow 
the instructor to add content, activities, and 
assessments, where appropriate. The content from 
the “original” base course is left unchanged. 

D1 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: User Interface 

D2: Clear and consistent navigation is present 
throughout the course. 

The course utilizes consistent and predictable 
navigation methods. Students can move logically 
and easily between areas of the course; color, 
graphics and icons are used to guide the student 
through the course; and a consistent look and feel 
exist throughout the course (consistent text, colors, 
bullets, and heading styles). Minimal training is 
required to navigate the course. 

D2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



D3: Rich media are provided in multiple formats 
for ease of use and access in order to address 
diverse student needs. 

Course makes maximum use of the robust 
capabilities of the online medium and makes these 
resources available by alternative means (video, 

CDs, podcasts). 

D3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



D4: Technology is used to help increase self- 
efficacy of students. 

Technology used in the course does not hinder the 
student's ability to accomplish the academic goals 
set forth by the syllabus. 

D4 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Technology Requirements and Interoperability 

D5: All technology requirements (including 
hardware, browser, software, etc.) are specified. 

All technology requirements (including hardware, 
browser, software, etc.) are identified in the course 
description or during the student registration 
process and specified to students before they begin 
the course. 

D5 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION D: TECHNOLOGY 

Element 

Further Explanation 

D6: Prerequisite skills, course tools, and course 
software are identified and appropriate in relation 
to the students and course. 

All prerequisite technology skills, software, and 
online tools necessary for the specific class are 
identified in the course description or during the 
registration process and are shared with students 
before they begin the course. Tools should be 
appropriate, necessary for teaching and/or 
enriching the lesson, cross-platform and free to the 
student (or built into the course). 

D6 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



D7: The course is designed to meet internationally 
recognized interoperability standards. 

Interoperability technical standards allow sharing 
content among different learning management 
systems and ensure sharing of questions, 
assessments and results with others. 

D7 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



D8: Copyright and licensing status, including 
permission to share where applicable, is clearly 
stated and easily found. 

Course developers or publishers clearly state the 
copyright and licensing status of all content, 
including permission to share where applicable. 
Copyright and licensing information should be 
readily available, understandable and standardized 
in terms of use. 

D8 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Accessibility 

D9: Course materials and activities are designed to 
provide appropriate access to all students. The 
course, developed with universal design principles 
in mind, conforms to the U.S. Section 504 and 
Section 508 provisions for electronic and 
information technology as well as the W3C’s Web 
Content Accessibility Guidelines (WCAG 2.0). 

Through the use of web accessibility evaluation 
tools, all web pages required for students to engage 
in online education (e.g., registration, library, 
course materials, grade retrieval) are validated to 
conform to accessibility standards. NIMAS is used 
to ensure textbooks and other instructional 
materials are accessible to the visually impaired. 

D9 Rating (1 = not applied, 2 = partially applied, 3 = applied): 




SECTION D: TECHNOLOGY 

Element 

Further Explanation 

Subsection: Resources and Materials 

DIO: Student information remains confidential, as 
required by the Family Educational Rights and 
Privacy Act (FERPA). 

Defined course procedures for reporting grade and 
student information comply with the Family 
Educational Rights and Privacy Act (FERPA) 
http://www.ed.gov/policy/gen/guid/fpco/ferpa/ 
indcx.html) posted within the course. 

DIO Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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SECTION E: COURSE EVALUATION AND SUPPORT ELEMENTS 

Element 

Further Explanation 

Subsection: Accessing Course Effectiveness 

E 1 : The course provider uses multiple ways of 
assessing course effectiveness. 

A combination of student, instructor, content 

reviewers may be used to evaluate the course for 
effectiveness. A variety of methods may be used 
including course evaluations, student completion 
rates, satisfaction surveys, peer review, teacher and 
student feedback, and student performance on in¬ 
course as well as state or national assessments. 
University researchers have been encouraged to 
conduct studies on the effectiveness of the course. 

El Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Course Updates 

E2: The course is evaluated using a continuous 
improvement cycle for effectiveness. The findings 
are used to improve and update the course content 
as needed. 

The provider indicates the frequency of course 
evaluations, whether reviews are conducted 
internally or externally, and how the provider uses 
evaluation results to improve courses. Courses 
should be reviewed to keep the content current, 
engaging, and relevant. 

E2 Rating (1 = not applied, 2 = partially applied, 3 = applied): 



Subsection: Instructor and Student Support 

E3: Technical support and course management 
assistance are provided to students, the course 
instructor, and the school coordinator. 

Online technical help and support should be 
available any time. If 24/7 support is not available, 
support hours are clearly posted within the course 
or on the online program’s website and a maximum 
response time is noted. Assistance may take the 
form of Frequently Asked Questions, training 
resources, mentors, or peer support. 

E3 Rating (1 = not applied, 2 = partially applied, 3 = applied): 
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